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REMARKS 

In the application, Claims 1-17 are pending and rejected. After due consideration of the 
Examiner's comments, the application has been amended as set forth above. In view of these 
amendments, Applicant submits that the claims are allowable over the prior art and requests that 
the Examiner issue a notice of allowance. 

Rejections under 35 U.S.C. §112 

The Examiner rejects claims 10 and 1 1 under 35 U.S.C. §1 12, 2 nd para, as being 
indefinite. Specifically, the Examiner asserts that the term "dirty" is a relative term that renders 
the claim indefinite. 

Applicant respectfully submits that the term "dirty" when used to describe data is not 
indefinite in that it would be recognized by those in the art as meaning incomplete, corrupted or 
other data that cannot be properly processed by a learning machine. Such a definition is 
provided in Pat. No. 6,157,921, column 8, line 13. This patent is incorporated in the present 
application by reference at page 7, line 20. See also the article by Hernandez and Stolfo entitled 
"Real-world Data is Dirty: Data Cleansing and The Merge/Purge Problem", Data Mining and 
Knowledge Discovery, Vol. 2, Issue 1, Jan. 1998, pp. 9-37. A copy of the abstract for this article 
is attached as Exhibit A. 

The Examiner rejects claim 17 under 35 U.S.C. §112, 2 nd para, as being indefinite. 
Specifically, the Examiner asserts that the dichotomy is not completed. 

The claim has been amended to address this rejection. 

Rejections under 35 U.S.C. §102 

The Examiner rejects claims 1-17 under 35 U.S.C. §102(a) as being anticipated by 
Cristianini and Shawe-Taylor {An Introduction to Support Vector Machines, Cambridge 
University Press, 2000) (hereinafter referred to as "CST"). 

The independent claims have been amended to include the limitation that the probability 
of the two classes be the same under the distribution given by the first eigenvector. This 
constraint is imposed by the second eigenvector for the purpose of maximizing the alignment. 
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Additional text has been added to the specification to clarify this point. The added text does not 
constitute new matter in that it is taken from provisional application Serial No. 60/272,391, to 
which this application claims priority and which has been incorporated by reference into this 
application. A copy of pages 9 and 10 of the provisional application with the relevant portion 
highlighted is attached hereto as Exhibit B. 

The Examiner alleges that claim 7, which in the originally-submitted claims first 
introduces the use of the second eigenvector to obtain alignment, is anticipated by CST, p. 156, 
lines 29-41. Applicant respectfully submits that the cited paragraph neither mentions alignment 
nor describes a concept similar to alignment. Rather, the paragraph discusses maximal margin 
classifiers and soft margin classifiers which are merely different forms of analysis that can be 
formed by support vector machines. Maximal margin classifiers are used with linearly separable 
data where there is no overlapping of the samples, where the classifier with the largest margin 
will provide the lowest risk. Soft margin classifiers have some overlapping classes and are 
effected by using slack variables that permit some variables to be misclassified. These classifiers 
have nothing to do with alignment, which is used to determine similarity within and between the 
clusterings of a set of points. Quoting from the specification at page 12, line 6, "Alignment 
captures the notion of a good clustering as achieving high similarity within the clusters and low 
similarity between them;" and at page 16, line 20, "This approach is directed to finding clusters 
that have minimal "scatter" around their mean. Among other appealing properties of the 
alignment is that this quantity is sharply concentrated around its mean." See also the discussion 
on page 4 of the publication by Marina Meila, "Data Centering in Feature Space", from the 9 th 
International Workshop on Artificial Intelligence and Statistics, Jan. 2003, attached hereto as 
Exhibit C. 

CST does not disclose kernel alignment. Further, CST does not disclose that the second 
eigenvector is used to select the best aligned kernels by imposing the constraint that the 
probability of the two classes be the same under the distribution given by the first eigenvector. 
Therefore, CST fails to disclose each and every element of the claimed invention as required 
under §102. Accordingly, Applicant respectfully requests that the prior art rejection be 
withdrawn. 
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Should the Examiner believe that prosecution of this application might be expedited by 
further discussion of the issues, he is invited to telephone the undersigned attorney for Applicant 
at the telephone number indicated below. 



PROCOPIO CORY HARGREAVES & SAVITCH LLP 
530 B Street 
Suite 2100 

San Diego, California 92101-4469 
Telephone: (760) 931-9703 (direct) 
Facsimile: (760)931-1155 
emm@procopio.com 

Docket No. 0171 [112092-000041] 
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Dated: January 27, 2005 
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Abstract 

The problem of merging multiple databases of information about common entities is 
frequently encountered in KDD and decision support applications in large commercial 
and government organizations. The problem we study is often called the Merge/Purge 
problem and is difficult to solve both in scale and accuracy. Large repositories of 
data typically have numerous duplicate information entries about the same entities 
that are difficult to cuD together without an intelligent "equational theory" that iden- 
tifies equivalent items by a complex, domain-dependent matching process. We have 
developed a system for accomplishing this Data Cleansing task and demonstrate its 
use for cleansing lists of names of potential customers in a direct marketing-type ap- 
plication. Our results for statistically generated data are shown to be accurate and 
effective when processing the data multiple times using different keys for sorting on 
each successive pass. Combing results of individual passes using transitive closure over 
the independent results, produces far more accurate results at lower cost. The system 
provides a rule programming module that is easy to program and quite good at finding 
duplicates especially in an environment with massive amounts of data. This paper 
details improvements in our system, and reports on the successful implementation for 
a "real-world" database that conclusively validates our results previously achieved for 
statistically generated data. 
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gration 
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Theorem 2 Similarly can lift done for *leave one out* cases: if, by removing 
a variable altogether, the function doe* not change much, then it i concentrated. 
NOTICE: SUR OF SUMS IS CON 0 KrY 1" HATED !!!!!! 
Consequence: concentration of spectrum 

This means that wc can trust the results from spectral graph Lhcory to give 
iih good hypotheses ... [CHUCK THIS AGAIN !] 

Wc could also use Andre's approach, with stability analysis for producing 
hounds ... 

5 Spectral Kernel Algorithms 

Wc will assume that we are given a datafiflt of m point** drawn from a set A' 
according to a fixed distribution '/>. In the Aiipervmed leaxuing caae wc will also 
assume we are given a vector y 6 {— I » + 1 } m (labels), and in the semisuper vised 
or transductWc raac that wc arc given a vector y C {1, *, -r-l} m , where * mans 
that there ia no label for a particular point. 

A Distribution on the Data S«t Cormider a graph G whose adjacency 
matrix i* K. Consider a random walk on the graph. Consider its stationary or 
ergodic distribution, it somehow measures the popularity of a given node, how 
much time a random walker will spend on it. This depends on the number, 
popularity and closeness of its neighbours. 
It is given by the first eigenvector of K. 

The distance between the lire t 2 eigenvalues gives information about the 
degree of connectivity of th* graph. Ft can be used to measure the amount of 
structure m Ui« data. 

Similarly, the entropy of the stationary distribution, can quantify it. 

NO'HCft: lifting can be used to explicitely adapt the Jirnt eigenvector, and 
hence adapt kernels. 

Remnrk One can perfoem data cleaning by considering the label as another 
feature. In thr« way, spotting isolated points would amount to spotting unusual 
combinations of x anil y. 

Maximum Alignment In [5] we proposed the quantity A = y*f\\j (called 
'alignment'} to measure the level of fitness between a kernel and a fixed labeling 
of the data. The purpose was to better cIloohc kernel parameters. An alternative 
- and perhaps more interesting - problem in to select the best aligned set of labels 
(pox»ibly .subject to some constraint), [can it be KP complete ? reduces 
to bi partitioning if replaces K with L] 

This is an absolute measure of a kernel: \t* second eigenvalue. It can be 
used Lo optimize kernel parameters 

We can look at an approximation of it, with eigentechniques: its value can 
be lower-bounded with the second eigenvalue. 

Consider the constraint C31.: )>2+P* — 2- Pi- We relax for the moment the 
requirement that y - — I or +1, and we ask that Ylv? = m ( we xy M »ee 



later why) £ + j>< = P** ^ iaL lh the lwo classes have equal probability under 
the distribution of the first eigenvector. [DTSCUSS THIS !] 

Under these, constraint*, the alignment, can Ik: maximized by »pecLial tech- 
nique*. 

Defining tj = M . the problem becomes: 
v(m) 

where «2 ' H tin*, mini mixer, the hccoikI eigenvector. 

That !H the secoiul eigenvector ruHxirniKcs the alignment under the constraint, 
and thresholding it provides with a labelling thai approximately maximizes 
alignment. The second eigenvalue gives the value of the alignment in this case, 
and is a lower hound under the true value of the optimal alignment. 

A very important remark. Now one has an 'ahMoluLc 1 measure of kernel 
alignment: the maximal alignment achieved on all possible labeling*. And this 
is equivalent to the second eigenvalue. So one can atao nine the kernel param- 
eters in a principled way, maybe by gradient, dc*«:eni., no to achieve maximal 
alignment. The thresh nidi ng of the. seeond eigenvector will then be the output. 

The constraint m not however jinn a technical problem: it actually has a very 
clear meaning: IT aEQUlHfcS THE PROBABILITY OF THE TWO CLASSES 
TO THE SAME UNDER THE DISTRIBUTION GTVfiN nr TTTF, FT71ST 
EIGENVECTOR ... (analogous to the requirement of a balanced split 
in the laplacian case, how to relax it 7) 

This gives us many natural algorithms; 

Unsupervised - clustering Tint choice of a kernel automatically defines 
two cIrhhck in (.he unlabeled datasct by means of the sign of the second eigenvec- 
tor, Su<:c:«nhcv<! eigenvector)* can be used for further parti tionings. A measure of 
the good uetw of a given duster is the second eigenvalue, or its aligment. Kernel 
parameters can be tuned to optimize it. Analogously can be done with the 
laplacian. 

The experiments describe the. performance of the clustering algorithm tested 
on WiHconHiri lireast Cancer and on Ionosphere data. In the experiments, the 
algorithm was used to split in two classe the data, then the aplil was compared 
with the labels, to measure the error. 

One can take also advantage of the information contained in the oilier ker- 
nels, as can be seen in the other plot. 

An algorithm for assessing the clustering power of a kerne! goes at follows: 

• build K 

• build L 

• compute* eigA 

• number of chiat.ers is number ol" ( appro x) zero eigenvalues 

Kor multiclasa cases, the alignment is denned, but one should replace y,-^ 
with Vi == Vj in the matrix to be compared vh K, Again, it would give a 
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Abstract 

This paper presents a family of methods for data 
translation in feature space, to be used in con- 
junction with kernel machines. The translations 
are performed using only kernel evaluations in 
input space. We use the methods to improve the 
numerical properties of kernel machines. Exper- 
iments with synthetic and real data demonstrate 
the effectivenes of data centering and highlight 
other interesting aspects of translation in feature 
space. 



1 Introduction 

Support vector machines (SVMs for short) classify data by 
mapping it into a high (possibly infinite) dimensional fea- 
ture space and constructing a maximum margin hyperplane 
to separate the classes in that space. Operations in the fea- 
ture space are rendered independent of its dimension by 
what is commonly called now the "kernel trick", the use 
of an efficiently computable kernel function for the scalar 
product in feature space. 

The SVM classifier is learned from data by means of the 
Gram matrix K consisting of the pairwise scalar products 
of the data points in feature space. If, in the feature space, 
the origin is far away from the convex hull of the data, then 
the elements of K have about the same value and, as a re- 
sult, the matrix K is ill-conditioned. Figure 1 illustrates 
such a situation, showing that the performance of the re- 
sulting classifier degrades. 

The present work sets out to correct this problem, by shift- 
ing the data such that the origin is located in the convex 
hull of the data. While this is almost trivial for a linear 
classifier, it is not so for non-linear SVMs where the data 
are mapped non-linearly into a high-dimensional space that 
is not explicitly represented. Thus, the challenge is to per- 
form the shift and to compute the resulting SVM using only 
"allowed" operations, that is applying the kernel to points 




Figure 1 : Original SVM classifier (dotted line) and classifier ob- 
tained after translating the origin in feature space by a 500 units 
(full line). The two classes are represented by and "x" re- 
spectively. The kernel used is the RBF kernel. 



in the input space. 

This paper presents kernel tricks that allow one to perform 
a large variety of origin translations in the feature space of 
a non-linear kernel machine. In its simplest version, this 
method of data centering in feature space has been in use 
for a long time (see e.g [4]). Here we show that one can 
formulate data centering in the form of criterion to be opti- 
mized by a translation in feature space, and that this trans- 
lation can be performed entirely by "allowed" kernel oper- 
ations. 

There has been previous work on adapting kernels to the 
data, notably that of [1, 5, 6]. The current idea differs from 
the above in that it does not affect the geometry of the prob- 
lem, it only affects its translation into a numerical task. 

An origin placed far away from the data is not the only 
cause of numerical problems in SVMs. Another com- 
mon problem is the inappropriate choice of kernel width 
(typical for RBF kernels) which can result into overfit- 
ting/underfitting [8]. A related problem known as "ridge 
effect" occurs in string kernels [13]: the data are almost or- 
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thogonal to each other in feature space. This can lead to 
both numerical problems and overfitting. While a transla- 
tion of the data may alleviate the numerical problems, the 
geometry of the classifier cannot be influenced by origin 
shift in feature space. Therefore we will not deal with these 
problems in the current paper. 

We start with a short introduction to support vector ma- 
chines, then we introduce a simple method of data center- 
ing in section 3 and explain how to perform classification 
with shifted support vectors in section 4. Next we present 
a general method of shifting in feature space in order to 
optimize some given centering criterion. Experiments are 
presented in section 6 and the discussion in section 7 con- 
cludes the paper. 

2 Support Vector Machines 

We start by briefly introducing the SVM; for more details 
the reader is invited to consult [8]. In a classification task, 
we are given a set of n data points {xi,X2, ■ • • x n }, ele- 
ments of an input space X. Each data point x* is labeled 
by yi e {±1}. The task is to use the data and labels to 
construct a classifier, i.e. to find a function / that predicts 
the label y of a new point x. 

Input space and feature space. We call the space of the 
original data the input space. The data are mapped into a 
space X called the feature space by a function <fi 



x = 4>{x) for all x e X 



(1) 



We assume that the data is linearly separable in X, mean- 
ing that a hyperplane that separates the two classes exists. 
The feature space is a Hilbert space whose dimension d is 
commonly much larger than n the number of data points 
and it can be infinity (e.g in the RBF kernel [8]). The in- 
put space need not be a Hilbert space, it can be any set. The 
trick that makes SVMs work is never to explicitly represent 
points in feature space or <f> itself. The SVM only makes 
use of scalar products of points in feature space, which are 
computed by the function 



K(x,z) = < <f>(x),<l>(z) > = < x y z > 



(2) 



called the kernel associated with <j>. It is assumed that 
K(x, z) can be computed efficiently for any pair of inputs. 
To be represent a scalar product, a symmetric kernel K is 
subject to the Mercer condition [8], namely that it induces 
a positive definite integral operator on X. 

Finding the optimal hyperplane. The separating hyper- 
plane is described by the equation < tu, x > +6 = 0, with 
w a vector in feature space and b a real number. The opti- 
mal hyperplane is found by solving an optimization prob- 



lem in the variables a*, i = 1 , 2, . . . n. 

n 

maxV(a) s.t. a* > 0 and Y^a^i — 0 (3) 

1=1 

with 

n ^ n n 

V ( a ) = £ ai " 2 Yl <*i<*jViVj < x U x j > w 

i=l i=l j=l 

The optimal hyperplane is then obtained from a i by 



w 



(5) 



& = V* ~^2 Q jV3 < X *> X 3 >• * 0r SOme * S t a » ^ °( 6 ) 

Note that although the optimization problem involves the 
data images in feature space, the data points enter V only 
via the pairwise scalar products < Xj,Xj >= K(xi,Xj) 
which can be computed using the kernel function. This is 
the celebrated "kernel trick" of support vector machines. 
The matrix 



(7) 



is called the Gram matrix 1 . Throughout the rest of the pa- 
per, we assume that a function SVM- SOLVER is given. 
The function SVM-Solver takes as input a Gram matrix 
K and a set of labels y returns the parameters 6, a*, i — 
l,...nofanSVM. 

Classifying with SVMs. When a new point x is pre- 
sented for classification, its label is computed by 



y = f(x) = sign 



^2aiyiK(x,Xi) + b 



(8) 



Note that the function / uses only kernel computations so it 
can be computed explicitly. If the kernel K is a non-linear 
function in each argument, then the resulting classifier / is 
a non-linear classifier. 

SVM extensions For the sake of simplicity, we have pre- 
sented here only the most basic version of SVM. Many 
other version exist that build upon the basics, some meant 
to deal with the case of non linearly separable data (C-SVM 
[3], J/-SVM [11]), others adapted for classfication from 
positive examples only [9], and others meant to deal with 
more than two classes [10]. All SVM versions cited here 
have in common the use of the Gram matrix as the only ve- 
hicle by which the data enter the SVM training. Therefore, 
the methods for data translation we present here should ap- 
ply to them as well. 

1 Wc shall use the same notation K for both the Gram matrix 
and the kernel; the distinction will be evident from the context. 



3 A simple centering method 

As shown in section 1 , if in feature space the origin lies 
far away from the data, then the matrix A" will have almost 
equal elements and will be ill-conditioned. 

How can we establish if, in the high-dimensional feature 
space, the origin lies "between" the classes or far-away 
from them? One way is to look at K{x^Xj) when data 
points i y j belong to different classes. If this scalar prod- 
uct is negative, it means that the points are seen from the 
origin under an obtuse angle, in other words the origin lies 
approximately between the two points. If we denote by K a 
the kernel representing the scalar product with the origin 
shifted in a 

K a (x, z) = <x y z> a = <x-a,z-a> (9) 

then we can define the optimal position of the origin to be 
the location a that minimizes 

Vi=i y>=-i 

Using 

<x,z> 0 = <ar, z > - < x,a> - < z y a> + <a,a> 

01) 

and letting n + (n ) denote the number of data points with 
+1( — 1) labels, we can rewrite J (a) as a quadratic criterion 
in a whose (unique) minimum is at 



Vi = l 



2n- 



(12) 



Thus the optimal a according to (10) is positioned halfway 
between the centers of gravity of the two classes in feature 
space. The kernel K a for this position of the origin may not 
be computable in closed form; nevertheless, we can obtain 
the Gram matrix K a necessary to solve the SVM optimiza- 
tion problem using only calls to the original kernel K. This 
is a consequence of the fact that a is a linear combination 
of data points in feature space. Denote 



, - J 0n + >- 1 . 
1 " I (2n-)"\ 



Vi = l 
Vi = -1 



(13) 



and 7 = [71 ... 7„] T , T = [7 . . . 7]. Then the "centered" 
Gram matrix is given by 

K a = [Kaixux^ij = K-T T K-KT+T T KT (14) 

Having obtained K a , a call to SVM-SOLVER(Jf a ,y) will 
output the parameters 6, i = 1, . . . n of an SVM. 

The optimal a obtained by the centering as in (12) may not 
belong to the data manifold in feature space, hence it is 
generally not representable by a point a in input space. In 



the next section we show that this fact does not preclude us 
us classifying new data points with the centered kernel. 

The criterion (10) and its solution can be generalized to 
problems that involve more than two classes. The details 
are presented in the longer version of the paper [7] 2 . 

Both the criterion (10) and its multiclass extension guaran- 
tee that the origin is contained within the convex hull of the 
data after the shift. They are very similar to the "standard" 
data centering method (e.g [4]) which moves the origin at 
the center of gravity of the data. In terms of the 7; coeffi- 
cients above, movig the origin to the center of gravity of all 
the data amounts to setting 7» = 

4 SVM classification with centered data 

We explain now how to perform classification with cen- 
tered data. Denote by 6, a, the output of SVM- 
SOLVER(A" a , y). According to equation (8), classifying 
a new data point x is done by 



f a (x) = sign 



^2<XiyiKa{x,Xi) + 6 



(15) 



Here, of course, we don't have K a (x y Xi) in closed form. 
This apparent obstacle can be overcome by using (3) and 
(1 1) to obtain (see [7] for details) 



f a (x) = sign 



Denote by 



^2 otiyiK(x, Xi) - ^2 a iVi <a>,Xi> +& 

* i 

(16) 



hi - < a, Xi > for i = 1, . . . , n, h = [hi /12 • - • h n ] T 

(17) 

By (12, 13) the values hi are 

n n 

h i = < Y,v x >> Xi > = Yl^ K ^^i) (i 8 ) 
j=i j=i 

Hence, a shift in the origin amounts to a constant correction 
term in the classifier, having the value 



Ab = ^^aiyajKixuXj) 
i=i >=i 



(19) 



Note that this quantity is in general non-zero for 7* = 1/n, 
i.e in the case of the "standard" centering method. More- 
over, if the origin is initially far away from the data, the 
value of A 6 can be quite large. 

Equation (16) has a geometric interpretation illustrated in 
figure 2. This result is a direct consequence of the fact that 
the optimal hyperplane used by the SVM to classify the 



2 Found at www . stat . Washington . edu/mmp 




Figure 2: The geometry of origin shift and its effect on b. 
O y a, x, are respectively the old origin, new origin and a data 
point; x\ a! arc the projection of x, a on the normal w to the sep- 
arating hyperplane (this direction is invariant to translation). The 
classification threshold b is the distance between the origin (old 
or new) to the hyperplane, and the change Ab due to the change 
in origin equals Oa the projection of Oa on w. The correction 
A6 accounts for the fact that the origin was shifted during train- 
ing while the new data are classified with the original, unshifted, 
kernel K . 



new points is invariant to translations in feature space. In 
other words, after shifting the origin we obtain the same 
classifier as we would from the original kernel with better 
numeric stability. The SVM classification with centered 
data can then be summarized as follows: 

1. Preprocessing: 

(a) Compute the Gram matrix K using definition (7) 

(b) Compute the centered Gram matrix K a by (9) 

(c) Compute the scalar products hi using (18) 

2. Training: Call SVM-Solver(JC 0 , y). This outputs 
6, a it % = l,...n. 

3. Postprocessing: 6 <— b — Ab 

4. Classification: Classify any new data points just as for an 
unmodified SVM classifier, i.e using 6, a*, t = 1, . . . n 
and the support vectors according to (8). 

In conclusion, centering in feature space only adds extra 
work in the SVM training phase, being essentially trans- 
parent in the classification phase. 

5 A general centering method 

We have shown how to perform an origin shift that opti- 
mizes the criterion (10). Now we proceed to generalize this 
method to optimizing any criterion J{K a ) that is a function 
of the Gram matrix only. Examples of such criteria are 

• Minimizing the sum of the cosines between all pairs 
of examples in different classes 

53 J2 cos ^ x ^ (20) 

Since the cosine between two points in feature space 
is a valid kernel (which places all the data points on 



the unit sphere), this criterion is equivalent to simple 
centering in a different feature space. Note however 
that the relationship between the two feature spaces is 
not straightforward. 

• Maximizing the kernel alignment of [6] defined as 

= Srf (21) 

n\K a \F 

with \K a \F being the Frobenius norm of K a . 

• Another apparently useful criterion is maximizing the 
"unnormalized" alignment 

y T Ky (22) 

At a closer inspection however, it can be seen that, 
unless n + = n~ = n/2 this criterion has a maximum 
foro oo in any direction so we do not recommend 
its usage. 

Let the centering criterion be 

max j(K a ) (23) 

We assume that J is a suitably smooth function of elements 
the Gram matrix, and in particular that its gradient is well 
defined. We show how to optimize J by gradient ascent. 

For this purpose, we first need to compute the gradient of 
J with respect to o. 

V a J(K a ) g, V.JT«(*i,*j) (24) 

From equation (11) 

V a K a {x u Zj) = -Xi-Xj+2a (25) 

The gradient is a vector in feature space and, by combining 
the two above formulae, one easily sees that the gradient 
is a linear combination of a and the data vectors. Taking a 
step in the direction V a J with step size r\ means 

a +- a + 77V fl J (26) 

The step <5 a = 7/V a J is itself a linear combination of o and 
the data vectors, hence 

$ a = 7oa + £7iZ; (27) 

i 

with 

70 = -|> 7< = -^£ad^' for<=1, 

(28) 

All the coefficients 7 above can be easily computed using 
only kernel evaluations. At each step of the iteration, we 
update: the Gram matrix K ay the scalar products hi =< 



n and the square length of a, ho =< The first set of experiments is performed on artificial data 

and aims to show that (1) drastic origin shifts in feature 
space harm the performance of an SVM classifier and (2), 
(29)that the simple centering algorithm is able to restore the ef- 
fects of the shift. We generated data normally distributed 



a,Xi >, t = 1,.. 
a, a > as follows: 

<X<, Xj > 0 +rf 0 = < Xi, Xj > a — < Xi, 6 a > — < Xj, 6 a 

+2 <6 a ,a > + < 6 a >6* > 
<6 a ,Xi> = 70 < Si, a > +y^7i' < x^Xj* > 



(30) 



around two concentric circles as in figure 1 and computed 



its Gram matrix K. Then we shifted the data in a ran- 

(3 1 )dom direction in feature space by a predetermined distance 

\a\ and computed the "shifted" Gram matrix K 8 . We then 

t a > centered the shifted data by the simple centering method 

described in section 3 and computed the "centered" Gram 

p2) matr i x Finally we trained an SVM using each of the 

... , 0 ^ x ^ i ^ x three^Gram matrices and evaluated it on test data from the 

^ , x ^ ^ ^,^x ^ ,i^ same distribution. 

<a + d 0l ii > = < a,ii > + < 6 a ,Xi > (33) 



<8 ai a> = 70 < a,a > +^7i < a,x» > 

i 

<6 a ,6a> = y^7»'7j' <Xj' i X j f >+2y^7j/ <Xj> 



+7o < a, a > 
< a + 8 ai a + 8 a 



With the previous notation for 7 and T we can summarize 
the gradient ascent algorithm as follows 

1. Initialize K a = K> h = 0, h 0 = 0. 

2. Compute 7, for * = 0, ... n by (28) 

3. Update K a , Sand h 0 by (29-33) 

4. Go to step 2 until convergence 

From the computational point of view, each gradient step 
requires order n 2 computations: order n 2 derivative eval- 
uations in step 2 and order n 2 update operations in step 3. 
This is of the same order of growth with one whole evalua- 
tion of the Gram matrix and affects only the training phase 
of the SVM classification. 

For large data sets, evaluating the whole K matrix is pro- 
hibitive and state-of-the-art SVM implementations evaluate 
only a subset of rows of K . In that case, the centering al- 
gorithms presented here would be prohibitive as well. We 
can easily fix this problem by using only a subsample of 
the data for centering, in a way similar to [12, 14]. For 
the simple centering method, we would sample e.g. n'/2 
data points from each class and represent a as the arith- 
metic mean of the subsample. This would still ensure that 
the new position of the origin falls inside the convex hull 
of the data, but the extra amount of computation per row of 
K a will be of order n' 2 . 

We can also use sampling to reduce the computational com- 
plexity of the general centering method In this case the 
solution is to redefine the optimal ity criterion J to involve 
only a submatrix of K at depending on a subset of n' «n 
points. While the solution may not work for any criterion, it 
is a reasonable approximation in the case of e.g optimizing 
the kernel alignment [12]. 

6 Experiments 

6.1 Shifting and recentering in feature space 

In the experiments we used the SVMLIB [2] source code, 
modified in order to accept a user-defined Gram matrix. 



The experiment was repeated 1 0 times with different sam- 
ples and shift directions for every value of |a|. We used the 
degree 2 polynomial kernel K(x, z) = (1 4- x T z) 2 and the 
RBF kernel K(x, z) = e" (i " i)2/ff2 . The training (test) set 
size was 300 (200) in all cases. The results are shown in 
figure 3. 

The second set of experiments was similar to the first, ex- 
cept that now we used real data sets from the UCI reposi- 
tory. The data set sizes are given in table 1 . The shift length 
was 1 000 for all data sets. For each data set, the experiment 
was run 10 times with different randomly sampled training 
sets. The results are shown in figure 4. 

From the two experiments we see first that a large shift is 
detrimental to classification performance. The recentered 
and original classifiers are almost identical for all the artifi- 
cial data experiments and for all but one of the real data sets 
(wdbc). This shows that recentering has indeed a restora- 
tive effect on data placed far away from the origin in feature 
space. The figures also show the effect on shifting and re- 
centering on kernel alignment: the alignment of the shifted 
kernel is practically 0 for the artificial data and drastically 
reduced for the real data. Recentering brings alignment 
back to near or above the original values. The number of 
support vectors, an indirect indicator of generalization per- 
formance, grows with the size of the shift in the polynomial 
kernel and for all but the largest shift in the RBF kernel, but 
drops elsewhere. The drop is very likely an artefact of the 
SVM-SOLVER software for extremely ill-posed problems 
(note that a shift of size 1000 is extremely large in the case 
of the RBF kernel). 

In the third set of experiments, we compared the centering 
method described in section 3 with shifting the origin in the 
center of weight of the data. To maximize the difference 
between the two methods, in these experiments the num- 
ber of examples in one class was 20 times larger than the 
number of examples in the other class. Testing was done 
on data generated from the same distribution as the train- 
ing data. The experimental setup mimicked the one for the 
first experiment. 
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Figure 3: SVM classification with original (dotted line), shifted (full line) and rccentered (dashed line) data for different values of the 
shift length \a\. The original data arc generated from the distribution shown in figure 1 (two concentric circles). These data arc shifted 
in a random direction by an amount |a| in feature space to obtain the shifted data. The shifted data are then recentered as in section 3 to 
obtain the recentered data. The figures depict the test error, kernel alignment and number of support vectors for the resulting SVMs, in 
the case of the polynomial degree 2 kernel (left) and of the RBF kernel (right). Results are averaged over 10 randomly sampled training 
sets of size 300. The original and centered SVMs are identical in all cases. The alignment of the shifted kernel is practically 0. 
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Figure 4: Shifting and recentering on real data sets: (a) test error, (b) alignment, (c) number support vectors. Circles represent original 
data, triangles - shifted data, squares - recentered data. The data sets are described in table 1. Each experiment was repeated 10 times 
with random direction shifts. The shift lenght \a\ was 1000 and the kernel was the RBF kernel in all cases. Note that the original and 
centered results are superimposed in a, c. 
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Figure 5: Alignment of the Gram matrices vs shift value for 4 
kernels: original (dotted), shifted (full), recentered by the center 
of weight method (dash-dot) and recentered by the simple method 
of section 3. 



Figure 6: SVM classification of original versus centered data in 
12 experiments with 10 data sets: (a) test error, (b) kernel align- 
ment. Each data point is the average of 10 random training/test 
splits. The data sets are described in table 1. 



For both centering methods, the recentered and the original 
classifier are essentially the same for the whole range of 
shifts. (The detailed results are in [7]). Therefore we can 
safely conclude that there is no practical difference between 
the two centering methods. 

An interesting aspect is revealed by the alignment plots in 
figure 5. Unlike figure 3 the alignment is maximum for 
the shifted data, while centering drastically reduces it. A 
quick analysis reveals the cause of this behavior: for a suf- 
ficiently large origin shift in feature space, the value of the 
alignment tends to 

A(K a ) — > (n+-rT) 2 /n 2 (34) 

In our case, n + is 20 times larger than n~ which yields the 
value A(K 8 ) = 0.82, in perfect agreement with the experi- 
ments. This strongly cautions us that optimizing the kernel 
alignment may not always produce the best classifier. 

6.2 Centering real data 

In this set of experiments, we applied the simple centering 
algorithm to real data. We computed the Gram matrices 
before and after centering, denoted by K and K a respec- 
tively), trained an SVM for each of them, and evaluated its 
performance on an independent test data set. The data sets, 
training and test set sizes, kernel types and parameters are 
given in table 1 . The SVM parameters were chosen so as to 
produce reasonable but not necessarily optimal classifica- 
tion results on the original data. This was done before the 
centering experiments, with one random training/validation 
split of the original data. 

The results are summarized in figure 6. Each point in the 
figure represents the average of 10 random training/test 
splits. The test error plot shows that, as expected, center- 
ing has no effect in most cases but it improves performance 
occasionally (here, in one case: the wdbc data with poly- 
nomial kernel). In none of the experiments did data cen- 
tering hurt performance. In most cases where performance 
wasn't improved, the SVM classifiers from the centerd and 



original data were virtually identical. 

The kernel alignment is slightly reduced in 2 of the 1 2 ex- 
periments and dramatically increased in 8 others. Again, 
we notice that improving the alignment per se does not nec- 
essarily guarantee an improvement in the classification per- 
formance. 

7 Discussion 

This paper has presented a family of methods for data shift- 
ing and centering in feature space. They can be used in 
conjunction with any kernel machine that incorporates the 
information from the data in a Gram matrix. Data center- 
ing in feature space does not, in theory, affect the result- 
ing classifier. We have shown that in practice, it can have 
a benefic effect when the Gram matrix is ill conditioned 
due to a poor position of the origin w.r.t the data in feature 
space. We have found no instances where data centering 
hurt the classification performance. 

When used for data centering, translation in feature space 
requires extra work only in the training stage of the SVM. 
The extra computations are of the order n 2 , but can be re- 
duced by standard sampling schemes. 

There have been many previous studies on kernel adapta- 
tion [6, 1, 5]. Our centering methods differ from the pre- 
vious as they do not attempt to obtain a more appropriate 
kernel and they do not change the geometry of the prob- 
lem. The aim of data centering is merely to hand the S VM- 
Solver a problem instance with better numerical proper- 
ties. 

Additionally, we have shown that the "standard" center- 
ing method present in the literature requires a correction 
term for b. The experimetns have also illustrated interest- 
ing aspects of the (lack of) relationship between the kernel 
alignment and classification performance in practice. In 
particular, translation in feature space can greatly change 
the alignment with no effect on the classifier perfromance. 




Table 1 : The data sets used in the experiments. 

Name Description # inputs # train # test SVM parameters 

cmc Contraceptive data, UCI repository (class I vs all others) 9 400 523 RBF 6* = 100, C = 1000 

glass Glass data, UCI repository (class 2 vs all others) 9 1 30 84 poly2, RBF cr 2 = 2, C = 1000 

wdbc Wisconsing breast cancer data, UCI repository 30 312 257 po!y2, RBF a 2 = 1000, C = 1000 

6ig\ab Handwritten digits from the USPS (digit a vs digit b 64 200 400 RBF a 1 = 1000, C = 1000 
where a,b€ {0,1,2,6}) 



We therefore experimented with factoring out the effects of 
origin translation by first centering the data and then maxi- 
mizing the alignment. The results are in the full paper. 

Here, the results of the theoretical investigation into data 
m translation in feature space have been used solely for data 

centering. We envisage however a more interesting realm 
of applications: shifting the data in order to obtain new ker- 
« nels, parametrized by the shift. Obtaining a new kernel by 

origin shifts is possible with composite kernel, such as the 
ones used in the classification of string data (see e.g [13]). 
If one shifts the element kernels before composition, then 
the operation amounts to more than a translation at the lev- 
erl of the composite kernel and it does affect the problem 
geometry. Preliminary experiments in this direction are al- 
ready under way. 
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